Content metadata enhancement of high dynamic range images

ABSTRACT

Image data is encoded for distribution in a lower bit-depth format. The image data has a range that is less than a maximum range and is mapped to the lower bit depth format using a mapping such that a ratio of a range of the lower bit depth representation to a maximum range of the lower bit depth representation is greater than a ratio of the range of the image data to a maximum range of the image data. Metadata characterizing the mapping is associated with lower bit depth representation. The metadata may be used downstream to reverse the mapping so that tonal detail is better reproduced.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationNo. 61/416,728 filed 23 Nov. 2010, hereby incorporated by reference inits entirety.

TECHNICAL FIELD

This disclosure addresses reduction of undesirable visually perceptibleartifacts when an image encoded with a bit depth m is displayed on adisplay having a bit depth n, where m<n. Metadata distributed with theimage characterizes the image's luminance range.

BACKGROUND

A display's bit depth corresponds to the number of levels of brightnessthat each of the display's pixels can reproduce or display. Higher bitdepth displays can reproduce more discrete levels of brightness. Forexample, a display having a bit depth of 1 may represent either one of2¹=2 levels of brightness in each pixel (e.g. each pixel can be ON orOFF). By contrast, in a display having a bit depth of 10, the displaymay be able to control each pixel to have one of 2¹⁰=1024 distinctlevels of brightness.

A color display may provide separate sub-pixels for each of a number ofprimary colors. For example, a display may provide pixels comprisingsub-pixels that are red, green and blue. The luminance and color of apixel can be controlled by varying the brightness of each sub-pixel.Greater bit-depth in the control of sub-pixels permits more discretelevels of luminance to be represented and also permits a color displayto display a larger numbers of different colors. With a bit-depth of onefor each sub-pixel one sub-pixel can be black or red, another sub-pixelcan be black or green, and the third sub-pixel can be black or blue. Inthis case, the pixel may represent any of 2^(1×)2^(1×)2¹=2³=8 colors.

A color display having a bit depth of 8 is capable of representing anyone of 2⁸=256 levels of brightness in each displayed sub-pixel. Forexample, a value of 0 may represent a value at the bottom of thesub-pixel's luminance range (typically black) and a value of 255 mayrepresent a value at the top of the sub-pixel's luminance range. Such adisplay can theoretically display any one of 2⁸·2⁸·2⁸=2²⁴=16,777,216colors in each pixel.

To facilitate their display, images are encoded using various codingschemes. Bit depth is an attribute of such schemes. If an image isencoded using a bit depth of 8 then each one of the encoded image'ssub-pixels (or pixels in the case of a monochrome image) may representany one of 2⁸=256 levels of brightness. If the same image is encodedusing a bit depth of 10 then each one of the encoded image's sub-pixels(or pixels in the case of a monochrome image) may represent any one of2¹⁰=1,024 levels of brightness. Thus, a higher bit depth provides afiner granularity within the luminance range of the pixels.

Some encoding schemes for color images do not directly specifybrightness of individual sub-pixels. For example, the LUV schemespecifies overall luminance (L) for a pixel and specifies color for thepixel using two chroma coordinate values U and V. Again, a greater bitdepth can be used to increase the number of distinct luminance stepsand/or colors that can be represented by the image data.

It is generally desirable that the luminance steps in a displayed imagebe small enough that a luminance difference of one step is not readilyperceptible to the human visual system (HVS). Steps larger than this canresult in visible artefacts such as banding, particularly in imageregions where the luminance is slowly varying. Since higher bit depthsmake possible finer steps, higher bit depths are desirable fordisplaying images having higher luminance ranges. However, imagesencoded with higher bit depths are larger (i.e. consume more computermemory or storage space and more bandwidth on communication links) thanimages encoded with lower bit depths, and accordingly require increasedcomputational processing time and resources before they can bedisplayed. Consequently, images are often encoded for distribution atlower than optimum bit depths, notwithstanding the availability ofdisplays having higher bit depth capabilities and the fact that imagesare often initially acquired at higher bit depths.

There is a need for practical and cost effective methods and apparatusfor distributing and reproducing images (both still and video images)having a desired image quality.

SUMMARY OF THE INVENTION

This invention provides methods and apparatus that may be applied in thedistribution, processing and display of image data. Aspects of theinvention provide: methods for encoding image data in a lower bit depthformat for distribution; methods for preparing distributed image datafor display and for displaying such distributed image data; apparatusfor encoding image data in a lower bit depth representation using avariable mapping; apparatus for processing lower bit depth image data toa higher bit depth using variable mappings; apparatus for displayingdistributed image data. The invention may be applied, for example indistribution and display of movies, television programming and othervideo content. The invention may, for example, be embodied intelevisions, video monitors, computer monitors, computer systems,special purpose displays and the like.

One aspect of the invention provides a method for distributing imagedata. The method comprises determining a range of the image data andmapping the image data to a reduced bit-depth format to produce a lowerbit-depth representation of the image data. The mapping is done with amapping such that a ratio of a range of the lower bit depthrepresentation to a maximum range of the lower bit depth representationis greater than a ratio of the range of the image data to a maximumrange of the image data. The method generates metadata characterizingthe mapping and associates the metadata with the lower bit depthrepresentation.

Another aspect of the invention provides a method for displaying images.The method comprises obtaining image data in a first format having afirst bit depth and corresponding metadata. Based on the metadata themethod generates a tone mapping for mapping the image data to a secondformat having a second bit depth greater than the first bit depth. Themethod then applies the tone mapping to map the image data to the secondformat and displays the second format data.

Another aspect of the invention provides image processing apparatuscomprising an image analyzer configured to determine a range of valuesin image data for an image in a first format and to generate a mappingfor the image from the first format to a second format having a lowerbit depth than the first format. The apparatus comprises a mapping unitconfigured to map the image from the first format to the second formataccording to the mapping and an encoding unit configured to encode thesecond format image data and metadata representing the mapping into adistribution format.

Further aspects of the invention and features of example embodiments ofthe invention are described below and illustrated in the accompanyingdrawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The appended drawings illustrate example non-limiting embodiments of theinvention.

FIG. 1 is a simplified block diagram depiction of a prior art imagingsystem in which n+-bit image data is encoded with a bit depth of m-bitsand distributed to a display having a bit depth n, where m<n.

FIG. 2 is a simplified schematic depiction of an imaging system in whichn+-bit depth image data is distributed over distribution paths having areduced bit depth. The example distribution paths include cable TV,satellite TV, disc (e.g. a DVD or a Blu-ray™ disc) and internet paths.

FIG. 3 schematically depicts display of bright and dark imagesrespectively on a dimmable display, in accordance with the prior art.

FIG. 4 schematically illustrates re-mapping a luminance range of imagedata.

FIG. 5 is a flow chart illustrating a method according to an exampleembodiment of the invention.

FIG. 6 is a simplified flowchart illustrating a method according toanother embodiment of the invention.

FIG. 7 graphically depicts shifting and rescaling luminance steps whileencoding image data to obtain improved image quality from image datapresented in a format having limited bit-depth. In the illustratedexample embodiment the bit depth is 5 bits.

FIG. 8 schematically depicts distributing image data by way of a m-bitdepth encoded image for display on a display capable of a bit depth ofn-bits according to an embodiment of the invention.

FIG. 9 schematically depicts apparatus according to an exampleembodiment of the invention.

DETAILED DESCRIPTION

Throughout the following description specific details are set forth inorder to provide a more thorough understanding to persons skilled in theart. However, well known elements may not have been shown or describedin detail to avoid unnecessarily obscuring the disclosure. Accordingly,the description and drawings are to be regarded in an illustrative,rather than a restrictive, sense.

Background

FIG. 1 illustrates an example case in which image data 10 having a bitdepth of at least n-bits (n+-bits) is converted to a lower bit depth ofm-bits and encoded for distribution by an image processor 14. m-bitsignal 15 is delivered to an image processor 16 (which may, for example,be internal to a display or external to the display). Image processor 16renders the signal in a bit depth of n-bits for display on display 12.

The bit depth of the image which is to be displayed can in practiceconstrain a display to an effective bit depth equal to that of theimage, notwithstanding the display's potentially higher bit depthcapability. For example, one approach to displaying an image encodedwith a bit depth of 8 bits on a display having a bit depth of 10 bits isto use the 8-bit image data as the most-significant 8 bits of a 10-bitdrive signal for the display. This has the advantage of using almost thefull luminance range of the display. However, the resulting image mayhave banding or other undesirable visually perceptible artifacts becausethe full bit depth of the display is not utilized and so steps betweenadjacent luminance levels may be visible. Such artifacts can beparticularly noticeable where the luminance range is large as, forexample, in the case where the display is a high-brightness, highdynamic range display. Using the 8-bit depth image data as the leastsignificant bits of the 10-bit drive signal for the display is alsoundesirable as the luminance range capability of the display is notfully or adequately utilized.

FIG. 2 depicts a high definition television (HDTV) 30 and a highdefinition computer monitor 32 connected to a computer 31. Computer 31and HDTV 30 are both connected to receive image data over variousdistribution media paths. HDTV 30 and computer monitor 32 are eachcapable of displaying image data having a bit depth of n bits. In eachcase, even if the image data originally had a larger bit depth, theimage data is distributed in a lower bit-depth format (e.g. m-bits). Inthis example, image data 20 initially has a bit depth of n-bits orgreater (n+-bits) and each distribution path includes at least onesection in which the image data is carried in a format having a bitdepth of m bits with m<n. m is not necessarily the same for alldistribution paths. Illustrated in FIG. 2 are an internet distributionpath 21, a disc (e.g. a DVD or a Blu-ray™ disc) distribution path 22, acable TV distribution path 23, and a satellite TV distribution path 24.

Image data 20 is obtained from a suitable source such as a camera 25.Internet distribution path 21 carries image data 20 in an m-bit depthformat across the internet. Disc distribution path 22 records image data20 in an m-bit depth format on a disc 26 that is played back on a discplayer 27. Cable television distribution path 23 carries image data 20in an m-bit depth format across a cable television system network 28 toa cable TV set-top box 29 which distributes lower bit depth (i.e. m-bit,where m<n) encoded versions of image data 20 to an n-bit backlit highdynamic range image display such as high definition television (HDTV)30, or high definition computer monitor 32. Satellite TV distributionpath 24 includes a satellite transmitter 33 which distributes an m-bitencoded versions of image data 20 to a satellite receiver 34. It will beunderstood that imaging system variants having fewer or more imagesources, fewer or more image processors, and fewer or more displays arepossible.

Some backlit displays have a “global dimming” capability whereby all ofthe lighting elements in the display's backlight can be simultaneouslydimmed or turned off. Global dimming can improve a display's luminancerange in comparison to an equivalent display having no dimmingcapability.

Some other backlit displays have a “local dimming” capability wherebyindividual lighting elements or groups of lighting elements in thedisplay's backlight can be selectively dimmed or turned off. Localdimming can significantly improve a display's dynamic range incomparison to an equivalent display having no dimming capability, orimprove a display's local contrast and simultaneous range in comparisonto an equivalent display having a global dimming capability.

FIG. 3 schematically depicts display of bright and dark imagesrespectively on a dimmable display, using a log(L) graphicalrepresentation, where L represents luminance in nits. High dynamic rangeimage data may, for example, specify luminance values ranging from 0.001to 10,000 nits, corresponding to bar 41 in FIG. 3. A display may becapable of displaying images having luminance values ranging from 0.1 to600 nits, for example, corresponding to bar 42 indicated in FIG. 3. Ifthe display has a dimming capability then the same display may also becapable of displaying dark images having luminance values ranging from0.01 to 60 nits, when the display's dimming capability is fullyutilized—corresponding to bar 43 indicated in FIG. 3.

FIG. 3 includes a histogram for one scene “Scene A” that may bedisplayed on the display. Scene A corresponds to a high dynamic rangeimage of a bright scene characterized by luminance values ranging from0.005 to 10,000 nits. The display's dimming capability is not used inorder to minimize reduction of the image's brighter characteristics whenthe Scene A image is displayed. Pixels of Scene A for which thespecified brightness exceeds 600 nits may be clipped to 600 nits orotherwise processed (e.g., scaled, tone-mapped, etc.) for display withinthe range indicated by bar 42. Similarly, pixels of Scene A for whichthe specified brightness is less than 0.1 nits may be clipped to 0.1nits or otherwise processed (e.g., scaled, tone-mapped, etc.) fordisplay within the range indicated by bar 42.

FIG. 3 also includes a histogram for another scene “Scene B” thatcorresponds to a high dynamic range image of a dark scene characterizedby luminance values ranging from 0.001 to 200 nits. The display'sdimming capability may be utilized to improve retention of the image'sluminance characteristics within an intermediate luminance rangeindicated by bar 44 when the Scene B image is displayed. Any pixels ofScene B for which the specified brightness exceeds 200 nits may beclipped to 200 nits or otherwise processed (e.g., scaled, tone-mapped,etc.) for display within the range indicated by bar 44. Similarly,pixels of Scene B for which the specified brightness is less than about0.03 nits may be clipped to 0.03 nits or otherwise processed (e.g.,scaled, tone-mapped, etc.) for display within the range indicated by bar44.

Content Metadata Image Enhancement

Embodiments of this invention redefine the mapping between image datavalues and corresponding luminance values. The redefinition may, forexample scale and/or apply offsets to this mapping. This approach may beapplied to obtain better quality images when image data is distributedin a lower bit-depth format. In some embodiments the redefined mappingmay be specified by metadata that is associated with the image data. Themetadata may be encoded in the image data, delivered with the image dataor delivered separately from the image data. In some embodiments, theredefined mapping is implemented at a display at least in part throughcontrol of global or local dimming

FIG. 4 provides an exaggerated illustration of how remapping of imagedata values to luminance levels can be used to improve image quality.Specifically, the set of lines 49 illustrates a possible mapping of 128image data values (a bit-depth of 7) to luminance values over a 500 nitluminance range. Lines 50 in FIG. 4 graphically depict a possiblemapping of 32 image data values (a bit-depth of 5) to luminance valuesover the same 500 nit luminance range. In each case a gamma of 2.4 hasbeen used to distribute the steps, as is common. The image data valuesfor lines 49 may, for example, be provided by a 7-bit binary number (bitdepth of 7). The image data values for lines 50 may, for example, beprovided by a 5-bit binary number (bit depth of 5). It can be seen thatthe steps between adjacent luminance values in lines 50 aresignificantly larger than the steps between adjacent luminance values inlines 49. Such large steps may be perceptible to the HVS and may resultin visible artifacts, such as banding, in an image.

Encoding image data originally in a higher-bit depth format (asexemplified by lines 49) into a lower bit-depth format (as exemplifiedby lines 50) results in multiple discrete luminance values from thehigher-bit depth image data being encoded as the same luminance value inthe lower bit-depth image data and in the steps between adjacent,distinctly representable, luminance values being increased.

In many images, image data values are concentrated toward higher- orlower luminance and are not spread uniformly over the entire range ofpossible image data values. For example, a dark image may have no or fewimage values above some threshold value and a bright image may have noor few image values below some threshold value. For such images,altering the mapping between image data values and luminance values canallow smaller steps between adjacent luminance values without increasingthe bit depth used to represent the image data values.

Consider the case of a dark image represented in higher-bit depth imagedata 49 in which no pixels have luminance values of over 100 nits.Mapping that image data to lower bit-depth image data 50 could result insignificant loss of detail since lower bit-depth image data 50 canrepresent only ¼ as many discrete brightness levels in the range of 0 to100 nits as higher bit-depth image data 49.

In FIG. 4, the set of lines 52 illustrate luminance values in a casewhere the same lower bit-depth image data values corresponding to lines50 have been re-mapped to correspond to luminance values in the range of0 to 100 nits. In this case, the remapping comprises scaling by a factorthat is less than 1 (a factor of 0.2 in the illustrated example). It canbe seen that the steps between adjacent luminance values are reduced ascompared to the steps between lines 50. Mapping higher-bit depth imagedata 49 in the range of 0 to 100 nits to lower bit-depth data using themapping exemplified by lines 52 can preserve detail that would have beenlost if one used a conventional mapping like that of lines 50.

In FIG. 4, the set of lines 54 illustrate luminance values in a casewhere the same image data values corresponding to lines 50 have beenre-mapped to luminance values in the range of 250 to 500 nits. In thiscase, the remapping includes both scaling and translation by an offset56. Again, it can be seen that the steps between adjacent luminancevalues are reduced as compared to the steps between lines 50. Wherehigher bit-depth image data 49 depicts an image consisting essentiallyof luminance values in the range of 250 to 500 nits then a mapping asexemplified by lines 54 may be used to preserve detail when encodingthat higher bit-depth image data in a lower bit-depth form.

Remapping of image data values to luminance values (or equivalent) maybe performed for an entire image or scene (group of images) or for localregions within an image or scene. If remapping is done for local regionsthen the local regions may be regions specified by a predetermined grid,regions identified by analysis of image data or the like.

Where an image or part of an image has been encoded into a lowerbit-depth representation using a special mapping then informationspecifying what mapping was used may be preserved and used subsequentlyto convert the image data back into a higher bit-depth representationfor display. The information characterizing the mapping may, forexample, be associated with the lower bit-depth image data as metadata.

FIG. 5 illustrates a method 55 that applies remapping as describedabove. Method 55 begins by analyzing an image or local region within animage in block 56. The analysis determines whether the luminance valuesin the image or local region fall (or mostly fall) within a limitedrange. If so, then block 57 generates a mapping that can be used to mapluminance values to image data values in a reduced bit-depthrepresentation of the image and vice versa. For example, if theluminance values are all in the range of 250 nits to 500 nits then themapping may correspond to lines 54 in FIG. 4. In some embodiments, block57 may perform clipping, compression, tone-mapping or like operations onimage data to cause the luminance values to fall within a limited range.In some embodiments block 57 selects a mapping most suited to an imageor local region within an image from among a number of predeterminedmappings. In some embodiments, block 57 generates a custom mapping basedon characteristics of the image or local region within the image.

Block 57 may generate a mapping based upon analysis of statisticalinformation regarding the image or local region within the image. Forexample, block 57 may generate a mapping based upon a histogram of theimage or local region within the image. The histogram may, for example,provide, for different luminance values or equivalent or differentranges of luminance values or equivalent the number of pixelscorresponding to the luminance value or range or luminance values. Froma histogram, one can determine whether: the image is primarily bright,primarily dark, or includes both bright and dark regions, whether theimage includes a broad range of luminance values, includes a narrowrange of luminance values, or the like. Such evaluation may be appliedto select an appropriate mapping.

In block 58 the mapping of block 57 is applied to map luminance values(or equivalents) from image data to image data values that are encodedat a reduced bit depth. In block 59 the encoded image data istransmitted over a transmission path. In block 60 metadata thatcharacterizes the mapping is also transmitted. Block 60 may comprise,for example, encoding the metadata in the image data itself, encodingthe metadata as part of a data package that includes the image data,transmitting the metadata together with the image data or transmittingthe metadata on a separate transmission path from the image data.

The metadata may specify a predetermined mapping and/or may provideparameters for a mapping (the parameters may, for example directly orindirectly specify an offset and/or a scaling factor and/or parametersof a linear or non-linear mapping function). In some embodiments,different mapping functions may be selected for images having differentcharacteristics. For example the mapping used for images made upprimarily of very bright highlights may have a different functional formthan the mapping function used for images made up primarily of darkshadows.

In block 62 the metadata is applied to recreate the mapping and in block64 the recreated mapping is applied to reformat the image data to agreater bit depth. In block 66 the reformatted image data is displayed.In method 55, blocks 56 to 58 may be performed, for example, at a sourceof image data, blocks 62 and 64 may be performed at a display or at adevice on a transmission path (such as a set top box or a transcoder,for example) upstream from a display.

It can be appreciated that application of method 55 to a series ofimages or local regions within an image can result in the same luminancevalues being encoded as different image data values for different onesof the images and/or for different local regions in the reducedbit-depth encoded image data. Application of method 55 to a series ofimages or local regions within an image may result in differentluminance values being encoded as the same image data values fordifferent ones of the images and/or for different local regions in thereduced bit-depth encoded image data.

FIG. 6 illustrates another example method 70 which can exploit dimmingdisplays. In method 70, metadata characterizing the luminance range of ahigh dynamic range image 71 can be distributed, with an m-bit encodedversion of the image, to a high dynamic range display 72 which may be anon-dimming display, a global dimming display or a local dimmingdisplay. Display 72 has an n-bit depth capability, where m<n. Image 71may be obtained from any one of a variety of image sources, and display72 may be any one of a variety of display types.

The luminance characteristic of image 71 is analyzed (FIG. 6, block 74)by a processor external to display 72. The processor may comprise anyone of a variety of image processors. If the block 74 analysisdetermines that image 71 is relatively bright (block 74 “bright” branch)then image 71 is m-bit encoded (block 76) utilizing a mapping in whichthe brightness of each pixel is mapped to one of 2^(m) image data levelswith a mapping such that the luminance values corresponding to the imagedata levels span image 71's luminance range (which, because image 71 hasno or only a few dark pixels, is smaller than the maximum luminancerange that image 71 could have).

If the block 74 analysis determines that image 71 is relatively dark(block 74 “dark” branch) then image 71 is m-bit encoded (block 78)utilizing a mapping in which the brightness of each pixel is mapped toone of 2^(m) image data levels with a mapping such that the luminancevalues corresponding to the image data levels span image 71's relativelydark luminance range (which, because image 71 has no or only a fewbright pixels, is also smaller than the maximum luminance range thatimage 71 could have). In general, image 71 is m-bit encoded utilizing amapping in which the luminance or equivalent for each pixel is mapped toone of 2^(m) image data values such that the luminance valuescorresponding to the image data values span image 71's luminance range.In some embodiments the luminance range of image 71 (or a region withinimage 71) excludes luminance values for outlying pixels. For example,the luminance range on which the mapping is based may not encompassluminance values below the X^(th) percentile for brightness and abovethe Y^(th) percentile for brightness. In some embodiments, the imageanalysis of block 74 may be configured such that a standard mapping isapplied in the event that block 74 determines that the image is neitherbright nor dark.

Metadata characterizing the mapping of block 76 (or from which themapping can be inferred) is generated in block 76A. Metadatacharacterizing the mapping of block 78 (or from which the mapping can beinferred) is generated in block 78A. In either case the metadata may,for example, characterize the luminance range of the image data beingencoded in the corresponding block 76 or 78.

In block 80 the m-bit encoded image data is distributed. In block 79 themetadata is distributed. In some embodiments block 79 comprisesembedding the metadata into the encoded image data so that distributingthe encoded image data also distributes the metadata. Alternatively, themetadata may be distributed in another manner In some embodiments theencoded image data and metadata are distributed via one or moredistribution channels like those described above in relation to FIG. 2.

In some embodiments, image data 71 has previously been analyzed(upstream from blocks 76 and 78) and includes or is associated withmetadata which directly or indirectly characterizes: the luminanceranges of images or regions within images represented in image data 71or mappings to be applied to those images or regions within images. Insuch embodiments, blocks 76 and 78 may comprise extracting or accessingthe previously-determined metadata and encoding the image data accordingto mappings based on that metadata. In some embodiments, block 74characterizes the luminance characteristics of a current image or regionwithin an image and the functions of blocks 76 and 78 are combined in ablock which encodes the image data for the current image or region usinga suitable mapping based upon the determination made in block 74.

Upon receipt of the m-bit encoded version of image 71 at a display orother device, method 70 proceeds differently depending upon whether thedevice is a legacy device which is not configured to process themetadata (block 82 “YES” branch) or a display that can process themetadata (block 82 “NO” branch).

For a legacy device the m-bit encoded version of image 71 is decoded butthe metadata is ignored (block 84). The decoded image is accordinglydisplayed without regard to the metadata, thus maintaining compatibilitywith legacy displays. If the device is equipped to process the metadata(block 82 “NO” branch) then the metadata is used to improve display ofimages in the encoded image data. In this example, the improved imagequality is obtained by using metadata to control a dimming capability ofdisplay 72. This is only one of a variety of ways to implement theinvention. For example, metadata associated with the m-bit encodedversion of image 71 may be applied to recreate the mapping utilized tom-bit encode image 71 and the recreated mapping may be applied to decodethe image data to n-bit depth. The invention is not limited to displayshaving dimming backlights or to the use of metadata to control backlightdimming in such displays.

In method 70 where the display 72 is a global dimming display (block 82“YES” branch and block 86 “GLOBAL” branch) then the m-bit encodedversion of image 71 is decoded and the associated metadata (which may,for example, be derived or extracted from the encoded image data) isprocessed to determine the image's luminance characteristic. Display72's backlight is then adjusted (block 88) in accordance with themetadata to best achieve the image's luminance characteristic, so thatthe m-bit encoded version of image 71 is mapped correctly to outputlight levels based on the metadata.

If display 72 is a local dimming display (block 82 “NO” branch and block86 “LOCAL” branch) then the m-bit encoded version of image 71 is decodedand the metadata associated with image 71 is processed to determine theimage's local luminance characteristics. Display 72's individualbacklight elements are then selectively adjusted (block 90) inaccordance with the metadata to best achieve the image's luminancecharacteristic.

In some embodiments, decoding of m-bit encoded image 71 and backlightadjustment are interdependent. For example, m-bit encoded image data 71may be decoded and the display's backlight may be adjusted such that thecombined re-mapping and backlight adjustment recreates or approximatesthe mapping utilized to m-bit encode image data 71 in step 76 or 78. Forinstance, where a mapping utilized in step 76 or step 78 comprises ascaling and a translation by an offset, m-bit encoded image 71 may bedecoded to apply the scaling and the brightness of the display'sbacklight may be adjusted to apply the offset. In method 70, for bothglobal dimming displays and local dimming displays, image quality isimproved because the initial mapping to the m-bit encoded image data isaltered to take into account global or local luminance characteristicssuch that steps between adjacent luminance levels are reduced, therebyreducing or eliminating perceptible artefacts resulting from largersteps between luminance levels.

FIG. 7 is similar to FIG. 4 but less exaggerated. FIG. 7 shows howlevels of m-bit encoded data may be mapped to different luminance levelsfor a bright image, a dim image, and an image that includes a full rangeof luminance values.

In embodiments where different mappings are applied to local regionswithin an image additional processing may be performed to avoidperceptible boundaries between different local regions (whether definedaccording to a predetermined grid or otherwise). The additionalprocessing may comprise, for example, one or more of:

-   -   spatial filtering;    -   dithering in the vicinity of region boundaries (e.g. in a        vicinity of a region boundary randomly or quasi-randomly        applying mappings from different adjacent regions to pixels);    -   blending between adjacent mappings (e.g. in the vicinity of a        region boundary computing pixel values for a plurality of        mappings associated with adjacent regions and interpolating        between the mappings so as to provide a smooth transition from        one mapping to another across the boundary region). In some        embodiments the blending may be performed according to spline        curves. The spline curves may be predefined or defined by        metadata.

FIG. 8 schematically depicts another example embodiment of theinvention. Original image data 100 is represented in FIG. 8 by ahistogram. Original image data 100 is in a format providing a higher bitdepth. For example, image data 100 may comprise data in a 16-bit integeror 16-bit float data format. Such formats permit a pixel to be assignedany one of a very large number of discrete luminance values within afull luminance range. Thus, steps between adjacent luminance values inimage data 100 can be small. The maximum and minimum luminance values oforiginal image data 100 span the maximum range 100A of luminance valuesthat can be represented by the data format.

Original image data 100 is windowed as indicated by 101 to providewindowed image data 102. Windowed image data 102 comprises luminancevalues within a reduced luminance range 102A that is smaller than thefull luminance range 100A that can be represented by the format of imagedata 100. Windowing 101 may comprise clipping luminance values toendpoints of reduced luminance range 102A, compressing luminance valuesthat are outside of reduced luminance range 102A into reduced luminancerange 102A, tone-mapping luminance values that are outside of reducedluminance range 102A into reduced luminance range 102A, a combination ofthese, or the like. In some embodiments, windowing 101 comprises tonecompression, such that within a particular luminance range, windowedimage data 102 comprises fewer different luminance values than originalimage data 100. Reduced luminance range 102A and the windowingoperations by which windowed image data 102 is obtained from image data100 may be selected to preserve details in original image data 100.Windowed image data 102 may still be represented in a format having arelatively high bit depth (which may be the same format as that oforiginal image data 100).

As indicated by 103 windowed image data 102 is mapped onto values in areduced bit-depth format to yield reduced bit-depth image data 104. Themapping applied by 103 is such that the reduced luminance range 102A ofwindowed image data 102 is mapped to a greater proportion of the range104A that can be represented by the reduced bit-depth format (i.e., themaximum range of the lower bit-depth representation) than would be thecase if the full luminance range 100A of image data 100 had been mappedto range 104A. In some embodiments the mapping of reduced luminancerange 102A is to the full extent of range 104A (or almost the fullextent of range 104A, for example, more than 85% or 90% or 95% or 98% ofrange 104A). Thus all or almost all of the distinct values that can berepresented by the reduced bit-depth format are used to represent valuesbetween the lower and upper ends of reduced luminance range 102A ofwindowed luminance data 102. Mapping 103 may comprise mapping n+-bitdepth image data to m-bit image data with m<n.

Metadata 107 characterizes the mapping used in mapping 103, and,optionally, the windowing performed at 101, such that a downstreamprocess can use metadata 107 to determine how to reverse the mappingperformed at 103, and, optionally, part or all of the windowingperformed at 101.

Reduced bit-depth data 104 may be distributed by any suitable transportmechanism. The lower precision of reduced bit-depth data 104 results inreduced bit-depth data 104 being smaller and more compressible thanoriginal image data 100. Metadata 107 is distributed with reducedbit-depth data 104. Reduced bit depth data 104 is distributed to adisplay or an intermediate device at which it is desired to convertreduced bit depth data 104 back into a higher bit-depth format fordisplay.

At the display or other device, reduced bit depth data is re-mapped asindicated by 105 to provide reconstructed data 106. Reconstructed data106 has a greater bit depth than reduced bit-depth data 104. Metadata107 is used by remapping step 105 to determine how the remapping oughtto be carried out to most closely recreate windowed data 102.Reconstructed data 106 can then be displayed.

In some embodiments, mapping 103 comprises mapping the top end ofwindowed image data 102 to the top end of reduced bit-depth image data104 and the bottom end of windowed image data 102 to the lower end ofreduced bit-depth data 104 and mapping intermediate values of windowedimage data 102 to corresponding intermediate values of reduced bit-depthimage data 104 according to a mapping that is uniform linearly orlogarithmically. In other embodiments intermediate values are mappedaccording to a non-uniform mapping such that discrete levels of thereduced bit-depth image data 104 are separated by smaller luminancesteps in luminance ranges having a lot of tone detail and separated bycorrespondingly larger luminance steps in luminance ranges in whichthere is not a lot of tone detail as compared to a uniform mapping. Theamount of tone detail present in any particular luminance sub-rangewithin luminance range 102A of windowed image data 102 may, for example,be estimated from an image histogram as an increasing function of thenumber of pixels in the luminance sub-range and the number of discreteluminance values those pixels have in the luminance sub-range. Thus,portions of the luminance range of windowed image data 102 in whichthere are few pixels and few distinct luminance values could be mappedto reduced bit-depth data 104 such that there are relatively large stepsbetween adjacent luminance values and portions of the luminance range ofwindowed image data 102 in which there are many pixels and many distinctluminance values could be mapped to reduced bit-depth data 104 such thatthere are relatively small steps between adjacent luminance values.

For example, the correspondence between image data values and luminancesin original image data 100 and windowed image data 102 may be a linearor logarithmic relationship. By contrast, the correspondence betweenimage data values and luminances in reduced bit-depth image data 104 mayfollow a nonlinear function such as a double s curve or a curveapproximated by a polynomial.

As an example, a high bit depth high dynamic range image may be providedas a 10-bit log-encoded signal having an expected response range of0.001 to 600 nits. The display may have a global or a locally dimmablebacklight, and an 8-bit liquid crystal display (LCD) panel having a700:1 contrast ratio and a peak brightness capability of 500 nits. Theluminance metadata characterizing one or more selected luminancecharacteristics of the image is derived as explained above. For example,the distribution of luminance values in the image can be used to derivean optimal tone curve corresponding to an 8-bit mapped version of theimage. The optimal tone curve may take into account the capabilities ofthe display so as to maintain the most detail in the image. Themetadata, e.g. the tone curve and black level are then sent to thedisplay along with the image data. Knowing the image's luminance rangefrom the metadata the display can map the image data into theappropriate luminance range at the higher native bit-depth of thedisplay.

In some embodiments for at least some images, the lower bit-depthencoded data received by a display (or other intermediate imageprocessing device) is directly mapped to bits of the higher bit-depthdrive for the display. For example, in some cases, 8-bit lower-bit depthdata may be mapped directly to the most significant bits of 10-bit imagedata for driving the display.

In contrast, a conventional prior art imaging system would typically usea predefined, invariant, mapping scheme to map the 10-bit log-encodedsignal to produce an 8-bit power/gamma encoded version of the image,which would then be displayed using fixed brightness and response. Asall of the mappings are fixed, dark scenes that are clear and detailedin the original image data input would be limited to fewer bits and morebrightness on the bottom end.

FIG. 9 schematically illustrates apparatus 110 according to an exampleembodiment of the invention. Apparatus 110 comprises an image datapreparation part 111A and an image data restoration part 111B. Imagedata preparation part 111A does not necessarily operate in real time(i.e. fast enough to process video frames at the display rate of thevideo frames) although it may do so. Image data preparation part 111Aprepares the image data for distribution and image data restoration part111B restores image data for display.

Image data preparation part comprises a data store 112 containingoriginal image data 114. Original image data 114 may comprise still orvideo images. In the following discussion it is assumed that originalimage data 114 comprises video data and the video data comprises aseries of video frames. Apparatus 110 comprises an image analysis module116. Image analysis module 116 determines a luminance range of a frame(or, in some embodiments, of a series of frames, or of a local regionwithin a frame or series of frames). Optionally image analysis module116 also evaluates whether conditions exist such that the frame may bewindowed to have a smaller luminance range without undesirabledegradation in image quality and, if so, the degree and/or type ofwindowing to be applied. In some embodiments image analysis module 116generates or obtains a histogram for the frame and performs analysis onthe histogram. The histogram may, for example, plot the number of pixelsin an image having each pixel value.

In the illustrated embodiment, apparatus 110 comprises a windowingmodule 118 that performs windowing on each frame in response to acontrol signal from image analysis module 116. A mapping unit 119 mapsthe windowed frames from windowing module 118 to a lower bit depth usinga mapping based upon the luminance range of each windowed frame. Themapping can vary from frame-to-frame. In some embodiments mapping unit119 provides a plurality of predetermined mappings. For example, mappingunit 119 may comprise a plurality of lookup tables that accept aluminance value (or equivalent) from the windowed frame as input andoutput a corresponding mapped value. Each lookup table may correspond toa different mapping. As another example, different mappings may bespecified by different parameter values used by a processor (a pluralityof sets of predetermined processor values may optionally be provided).As another example, different mappings may be provided by differentlogic pathways and/or by different software modules.

An encoder 120 encodes the lower bit depth image data together withmetadata in a distribution format 122. Metadata may be encoded intostandard metadata paths such as supplemental enhancement information(SEI) messages, broadcast teletext, or the like. The image data indistribution format 122 may then be distributed, for example, by writingit to a disc or other distribution medium, broadcasting on a cabletelevision or other broadcast medium, file transfer, streaming datatransfer or the like. The distribution channel is indicated by 125 inFIG. 9. FIG. 9 illustrates two architectures. In one case, thedistributed image data is processed by a processor 130A internal to adisplay 132A. In another case, the distributed image data is processedby an external processor 130B external to a display 132B.

Displays 132A and 132B each have a bit depth greater than that of thedistributed image data. Processors 130A and 130B each comprise a decoder134 that decodes the distributed image data and a metadata extractor 136that extracts the metadata from the distributed image data. A mapper 140maps the distributed image data to the bit depth of the displayaccording to a mapping set according to the metadata. The mapper may,for example, comprise a set of lookup tables or logic paths thatimplement different predetermined mappings, a logic path that implementsmappings according to supplied parameter values or control inputs, aprocessor that implements a mapping algorithm that performs mappingsaccording to supplied parameter values or control inputs, a processorthat executes one of a plurality of predetermined mapping algorithms, orthe like.

Conclusion

Systems and modules described herein may comprise software, firmware,hardware, or any combination(s) of software, firmware, or hardwaresuitable for the purposes described herein. Software and other modulesmay reside on servers, workstations, personal computers, computerizedtablets, PDAs, and other devices suitable for the purposes describedherein. In other words, the software and other modules described hereinmay be executed by a general-purpose computer, e.g., a server computer,wireless device or personal computer. Those skilled in the relevant artwill appreciate that aspects of the system can be practiced with othercommunications, data processing, or computer system configurations,including: Internet appliances, hand-held devices (including personaldigital assistants (PDAs)), wearable computers, all manner of cellularor mobile phones, multi-processor systems, microprocessor-based orprogrammable consumer electronics, set-top boxes, network PCs,mini-computers, mainframe computers, and the like. Indeed, the terms“computer,” “server,” “host,” “host system,” and the like are generallyused interchangeably herein, and refer to any of the above devices andsystems, as well as any data processor. Furthermore, aspects of thesystem can be embodied in a special purpose computer or data processorthat is specifically programmed, configured, or constructed to performone or more of the computer-executable instructions explained in detailherein.

Software and other modules may be accessible via local memory, via anetwork, via a browser or other application in an ASP context, or viaother means suitable for the purposes described herein. Examples of thetechnology can also be practiced in distributed computing environmentswhere tasks or modules are performed by remote processing devices, whichare linked through a communications network, such as a Local AreaNetwork (LAN), Wide Area Network (WAN), or the Internet. In adistributed computing environment, program modules may be located inboth local and remote memory storage devices. Data structures describedherein may comprise computer files, variables, programming arrays,programming structures, or any electronic information storage schemes ormethods, or any combinations thereof, suitable for the purposesdescribed herein. User interface elements described herein may compriseelements from graphical user interfaces, command line interfaces, andother interfaces suitable for the purposes described herein.

Image processing and processing steps as described above may beperformed in hardware, software or suitable combinations of hardware andsoftware. For example, such image processing may be performed by a dataprocessor (such as one or more microprocessors, graphics processors,digital signal processors or the like) executing software and/orfirmware instructions which cause the data processor to implementmethods as described herein. Such methods may also be performed by logiccircuits which may be hard configured or configurable (such as, forexample logic circuits provided by a field-programmable gate array“FPGA”).

Certain implementations of the invention comprise computer processorswhich execute software instructions which cause the processors toperform a method of the invention. For example, one or more processorsin a video workstation, set top box, display, transcoder or the like mayimplement methods as described herein by executing software instructionsin a program memory accessible to the processors.

The invention may also be provided in the form of a program product. Theprogram product may comprise any non-transitory medium which carries aset of computer-readable signals comprising instructions which, whenexecuted by a data processor, cause the data processor to execute amethod of the invention. Program products according to the invention maybe in any of a wide variety of forms. The program product may comprise,for example, physical media such as magnetic data storage mediaincluding floppy diskettes, hard disk drives, optical data storage mediaincluding CD ROMs, DVDs, electronic data storage media including ROMs,flash RAM, hardwired or preprogrammed chips (e.g., EEPROM semiconductorchips), nanotechnology memory, or the like. The computer-readablesignals on the program product may optionally be compressed orencrypted. Computer instructions, data structures, and other data usedin the practice of the technology may be distributed over the Internetor over other networks (including wireless networks), on a propagatedsignal on a propagation medium (e.g., an electromagnetic wave(s), asound wave, etc.) over a period of time, or they may be provided on anyanalog or digital network (packet switched, circuit switched, or otherscheme).

Where a component (e.g. a software module, processor, assembly, device,circuit, etc.) is referred to above, unless otherwise indicated,reference to that component (including a reference to a “means”) shouldbe interpreted as including as equivalents of that component anycomponent which performs the function of the described component (i.e.,that is functionally equivalent), including components which are notstructurally equivalent to the disclosed structure which performs thefunction in the illustrated exemplary embodiments of the invention.

Unless the context clearly requires otherwise, throughout thedescription and the claims, the words “comprise,” “comprising,” and thelike are to be construed in an inclusive sense, as opposed to anexclusive or exhaustive sense; that is to say, in the sense of“including, but not limited to.” As used herein, the terms “connected,”“coupled,” or any variant thereof, means any connection or coupling,either direct or indirect, between two or more elements; the coupling ofconnection between the elements can be physical, logical, or acombination thereof. Additionally, the words “herein,” “above,” “below,”and words of similar import, when used in this application, shall referto this application as a whole and not to any particular portions ofthis application. Where the context permits, words in the above DetailedDescription using the singular or plural number may also include theplural or singular number respectively. The word “or,” in reference to alist of two or more items, covers all of the following interpretationsof the word: any of the items in the list, all of the items in the list,and any combination of the items in the list.

The above detailed description of examples of the technology is notintended to be exhaustive or to limit the system to the precise formdisclosed above. While specific examples of, and examples for, thesystem are described above for illustrative purposes, various equivalentmodifications are possible within the scope of the system, as thoseskilled in the relevant art will recognize. For example, while processesor blocks are presented in a given order, alternative examples mayperform routines having steps, or employ systems having blocks, in adifferent order, and some processes or blocks may be deleted, moved,added, subdivided, combined, and/or modified to provide alternative orsubcombinations. Each of these processes or blocks may be implemented ina variety of different ways. Also, while processes or blocks are attimes shown as being performed in series, these processes or blocks mayinstead be performed in parallel, or may be performed at differenttimes.

The teachings of the technology provided herein can be applied to othersystems, not necessarily the system described above. The elements andacts of the various examples described above can be combined to providefurther examples. Aspects of the system can be modified, if necessary,to employ the systems, functions, and concepts of the various referencesdescribed above to provide yet further examples of the technology.

These and other changes can be made to the system in light of the aboveDetailed Description. While the above description describes certainexamples of the system, and describes the best mode contemplated, nomatter how detailed the above appears in text, the system can bepracticed in many ways. Details of the system and method for classifyingand transferring information may vary considerably in its implementationdetails, while still being encompassed by the system disclosed herein.As noted above, particular terminology used when describing certainfeatures or aspects of the system should not be taken to imply that theterminology is being redefined herein to be restricted to any specificcharacteristics, features, or aspects of the system with which thatterminology is associated. In general, the terms used in the followingclaims should not be construed to limit the system to the specificexamples disclosed in the specification, unless the above DetailedDescription section explicitly defines such terms. Accordingly, theactual scope of the system encompasses not only the disclosed examples,but also all equivalent ways of practicing or implementing thetechnology under the claims.

From the foregoing, it will be appreciated that specific examples ofsystems and methods have been described herein for purposes ofillustration, but that various modifications may be made withoutdeviating from the spirit and scope of the invention. Those skilled inthe art will appreciate that certain features of embodiments describedherein may be used in combination with features of other embodimentsdescribed herein, and that embodiments described herein may be practisedor implemented without all of the features ascribed to them herein. Suchvariations on described embodiments that would be apparent to theskilled addressee, including variations comprising mixing and matchingof features from different embodiments, are within the scope of thisinvention.

As will be apparent to those skilled in the art in the light of theforegoing disclosure, many alterations, modifications, additions andpermutations are possible in the practice of this invention withoutdeparting from the spirit or scope thereof. The embodiments describedherein are only examples. Other example embodiments may be obtained,without limitation, by combining features of the disclosed embodiments.It is therefore intended that the following appended claims and claimshereafter introduced are interpreted to include all such alterations,modifications, permutations, additions, combinations andsub-combinations as are within their true spirit and scope.

What is claimed is:
 1. A method for distributing image data, the methodcomprising: determining a range of the image data; mapping the imagedata to a reduced bit-depth format to produce a lower bit-depthrepresentation of the image data with a mapping such that a ratio of arange of the lower bit depth representation to a maximum range of thelower bit depth representation is greater than a ratio of the range ofthe image data to a maximum range of the image data; generating metadatacharacterizing the mapping; associating the metadata with the lower bitdepth representation; and identifying a portion of the range of theimage data containing greater tone detail and generating the mappingsuch that the identified portion maps to a portion of the lower bitdepth representation that occupies a larger proportion of the maximumrange of the lower bit depth representation than the proportion of therange of the image data occupied by the identified portion.
 2. A methodaccording to claim 1 wherein associating the metadata with the lower bitdepth representation comprises encoding the metadata and the lower bitdepth representation to provide a distribution format comprising boththe metadata and the lower bit depth representation.
 3. A methodaccording to claim 1 comprising performing a tone compression prior tomapping the image data.
 4. A method according to claim 1 comprisingrepeating the method for a plurality of frames of a video wherein themapping is different for different ones of the frames.
 5. A methodaccording to claim 1 comprising distributing the lower bit depthrepresentation and metadata for display.
 6. A method according to claim1 wherein distributing the metadata and lower bit depth representationcomprises writing the metadata and lower bit depth representation onto anon-transitory distribution medium.
 7. A method according to claim 1wherein the lower bit depth representation has a bit depth of 9 or lessand the image data has a bit depth of 10 or more.
 8. A method fordisplaying images, the method comprising: obtaining image data in afirst format having a first bit depth and corresponding metadata; basedon the metadata, generating a tone mapping for mapping the image data toa second format having a second bit depth greater than the first bitdepth; applying the tone mapping to map the image data to the secondformat; and displaying the second format data; wherein the image data ina first format comprises portions of an image represented in the imagedata containing greater tone detail occupying a larger proportion of arange of the image data than a proportion of a range of original imagedata occupied by corresponding image portions in an original image fromwhich the image data in a first format was derived.
 9. Image processingapparatus comprising: an image analyzer configured to determine a rangeof values in image data for an image in a first format and to generate amapping for the image from the first format to a second format having alower bit depth than the first format; a mapping unit configured to mapthe image from the first format to the second format according to themapping; an encoding unit configured to encode the second format imagedata and metadata representing the mapping into a distribution format;the mapping unit further configured to identify a portion of the rangeof the image data containing greater tone detail and generate themapping such that the identified portion maps to a portion of the lowerbit depth representation that occupies a larger proportion of themaximum range of the lower bit depth representation than the proportionof the range of the image data occupied by the identified portion. 10.Apparatus according to claim 9 wherein a ratio of a range of the lowerbit depth representation to a maximum range of the lower bit depthrepresentation is greater than a ratio of the range of the image data toa maximum range of the image data.
 11. Image processing apparatuscomprising: a decoder configured to decode image data encoded at a firstbit depth; and, a mapping unit configured to map the image data to ahigher bit depth representation according to a variable mapping, thevariable mapping set according to metadata associated with the imagedata; wherein the image data encoded at a first bit depth comprisesportions of an image represented in the image data containing greatertone detail occupying a larger proportion of a range of the image datathan a proportion of a range of original image data occupied bycorresponding image portions in an original image from which the imagedata was derived.
 12. Apparatus according to claim 11 comprising adisplay connected to display images represented by the higher bit depthrepresentation.
 13. Apparatus according to claim 12 wherein the decoderis configured to extract the metadata from the image data.
 14. Apparatusaccording to claim 11 comprising a plurality of predefined mappingswherein the apparatus is configured to select one of the predefinedmappings based on the metadata.
 15. Apparatus according to claim 11comprising a plurality of lookup tables each providing a mapping betweenthe image data and the higher bit depth representation wherein themapping unit is configured to look up values of the higher bit depthrepresentation in one of the lookup tables.